{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第四章 句子滑窗检索"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ In Answer Relevance, input prompt will be set to __record__.main_input or `Select.RecordInput` .\n",
      "✅ In Answer Relevance, input response will be set to __record__.main_output or `Select.RecordOutput` .\n",
      "✅ In Context Relevance, input prompt will be set to __record__.main_input or `Select.RecordInput` .\n",
      "✅ In Context Relevance, input response will be set to __record__.app.query.rets.source_nodes[:].node.text .\n",
      "✅ In Groundedness, input source will be set to __record__.app.query.rets.source_nodes[:].node.text .\n",
      "✅ In Groundedness, input statement will be set to __record__.main_output or `Select.RecordOutput` .\n"
     ]
    }
   ],
   "source": [
    "import utils\n",
    "\n",
    "import os\n",
    "import openai\n",
    "openai.api_key = utils.get_openai_api_key()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "读取数据库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index import SimpleDirectoryReader\n",
    "\n",
    "documents = SimpleDirectoryReader(\n",
    "    input_files=[\"data/人工智能.pdf\"]\n",
    ").load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'> \n",
      "\n",
      "7 \n",
      "\n",
      "<class 'llama_index.schema.Document'>\n",
      "Doc ID: b03a0e50-2e8a-49bf-82f0-4a3909364809\n",
      "Text: 2/2/24, 2:43 PM ⼈⼯智能  - 维基百科，⾃由的百科全书\n",
      "https://zh.wikipedia.org/wiki/ ⼈⼯智能 2/13“⼈⼯智能”的各地常⽤名称 中国⼤陆⼈⼯智能 台湾⼈⼯智慧\n",
      "港澳⼈⼯智能 新⻢⼈⼯智能、⼈⼯智慧 ⽇韩⼈⼯知能 越南智慧⼈造 [展开] [展开] [展开] [展开] [展开] [展开]⼈⼯智能系列内容\n",
      "主要⽬标 实现⽅式 ⼈⼯智能哲学 历史 技术 术语⼈⼯智能（英语：artiﬁcial intelligence ，缩写为\n",
      "AI）亦称机器智能，指由⼈制造出来的机器所表现出来的智能。通常⼈⼯\n",
      "智能是指⽤普通计算机程序来呈现⼈类智能的技术。该词也指出研究这样的智能系统是否能够实现，以及如何实现。同 时，通过 医学 、神经科学\n",
      "、机器⼈学 及...\n"
     ]
    }
   ],
   "source": [
    "print(type(documents), \"\\n\")\n",
    "print(len(documents), \"\\n\")\n",
    "print(type(documents[0]))\n",
    "print(documents[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index import SimpleDirectoryReader\n",
    "\n",
    "documents_en = SimpleDirectoryReader(\n",
    "    input_files=[\"data/eBook-How-to-Build-a-Career-in-AI.pdf\"]\n",
    ").load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'> \n",
      "\n",
      "41 \n",
      "\n",
      "<class 'llama_index.schema.Document'>\n",
      "Doc ID: 5ab262c9-a207-4f2d-9513-fc4d5c350cf5\n",
      "Text: PAGE 1Founder, DeepLearning.AICollected Insights from Andrew Ng\n",
      "How to  Build Your Career in AIA Simple Guide\n"
     ]
    }
   ],
   "source": [
    "print(type(documents_en), \"\\n\")\n",
    "print(len(documents_en), \"\\n\")\n",
    "print(type(documents_en[0]))\n",
    "print(documents_en[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里通过将 documents 中各个文档的文本连接成一个字符串，然后创建了一个 Document 实例，该实例代表了整个文档集合。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index import Document\n",
    "\n",
    "document = Document(text=\"\\n\\n\".join([doc.text for doc in documents]))\n",
    "document_en = Document(text=\"\\n\\n\".join([doc.text for doc in documents_en]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 将中文标点符号替换成英文标点符号，方便后续处理\n",
    "# 如果是英文文档，可以跳过这一步\n",
    "# 不处理的话，会导致无法正确切分中文句子，会影响后续sentence_window的大小，导致输入长度大于gpt-3.5-turbo的最大限制\n",
    "document.text=document.text.replace('。','. ')\n",
    "document.text=document.text.replace('！','! ')\n",
    "document.text=document.text.replace('？','? ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 一、句子滑窗检索设置"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建了一个名为 node_parser 的解析器对象，指定了窗口大小为3，原始文本元数据键被设置为``original_text``。这样创建的解析器可以用于从文本中提取节点"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.node_parser import SentenceWindowNodeParser\n",
    "\n",
    "# create the sentence window node parser w/ default settings\n",
    "node_parser = SentenceWindowNodeParser.from_defaults(\n",
    "    window_size=3,\n",
    "    window_metadata_key=\"window\",\n",
    "    original_text_metadata_key=\"original_text\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义一个中文文本字符串  \n",
    "使用 node_parser 的 get_nodes_from_documents 方法从提供的文本中提取节点。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"你好. 你怎么样? 我很好!  \"\n",
    "\n",
    "nodes = node_parser.get_nodes_from_documents([Document(text=text)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_en = \"hello. how are you? I am fine!  \"\n",
    "\n",
    "nodes_en = node_parser.get_nodes_from_documents([Document(text=text_en)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "每个单独的词"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['你好. ', '你怎么样? ', '我很好!  ']\n",
      "['hello. ', 'how are you? ', 'I am fine!  ']\n"
     ]
    }
   ],
   "source": [
    "print([x.text for x in nodes])\n",
    "print([x.text for x in nodes_en])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "原整句"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "你好.  你怎么样?  我很好!  \n",
      "hello.  how are you?  I am fine!  \n"
     ]
    }
   ],
   "source": [
    "print(nodes[1].metadata[\"window\"])\n",
    "print(nodes_en[1].metadata[\"window\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"你好. 吧台. 猫狗. 老鼠\"\n",
    "text_en2 = 'hello. bar. cat. dog. mouse.'\n",
    "\n",
    "nodes = node_parser.get_nodes_from_documents([Document(text=text)])\n",
    "nodes_en2 = node_parser.get_nodes_from_documents([Document(text=text_en2)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['你好. ', '吧台. ', '猫狗. ', '老鼠']\n",
      "['hello. ', 'how are you? ', 'I am fine!  ']\n"
     ]
    }
   ],
   "source": [
    "print([x.text for x in nodes])\n",
    "print([x.text for x in nodes_en])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "你好.  吧台.  猫狗. \n",
      "hello.  bar.  cat. \n"
     ]
    }
   ],
   "source": [
    "print(nodes[0].metadata[\"window\"])\n",
    "print(nodes_en2[0].metadata[\"window\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 创建索引"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 `OpenAI` 的 `GPT-3.5-turbo` 模型创建了一个语言模型的实例，设置了温度参数为0.1。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms import OpenAI\n",
    "\n",
    "llm = OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 `ServiceContext.from_defaults` 方法创建了一个 `ServiceContext` 对象，该对象包含了用于索引构建的服务相关的上下文信息，包括语言模型、嵌入模型以及节点解析器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index import ServiceContext\n",
    "\n",
    "sentence_context = ServiceContext.from_defaults(\n",
    "    llm=llm,\n",
    "    embed_model=\"local:BAAI/bge-small-zh-v1.5\",\n",
    "    node_parser=node_parser,\n",
    ")\n",
    "\n",
    "sentence_context_en = ServiceContext.from_defaults(\n",
    "    llm=llm,\n",
    "    embed_model=\"local:BAAI/bge-small-en-v1.5\",\n",
    "    node_parser=node_parser,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 `VectorStoreIndex.from_documents` 方法创建了一个 `VectorStoreIndex` 对象，该对象用于存储和检索与文档相关的向量信息。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index import VectorStoreIndex\n",
    "\n",
    "sentence_index = VectorStoreIndex.from_documents(\n",
    "    [document], service_context=sentence_context\n",
    ")\n",
    "\n",
    "from llama_index import VectorStoreIndex\n",
    "\n",
    "sentence_index_en = VectorStoreIndex.from_documents(\n",
    "    [document_en], service_context=sentence_context_en\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将创建的索引持久化到指定目录`（\"./sentence_index\"）`。这样做可以在之后的运行中重新加载索引，而不必重新构建。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_index.storage_context.persist(persist_dir=\"./sentence_index\")\n",
    "sentence_index_en.storage_context.persist(persist_dir=\"./sentence_index_en\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "检查索引文件是否存在，如果不存在则重新构建,如果存在，它将使用 `load_index_from_storage` 方法从已有的索引文件中加载索引，而不是重新构建。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This block of code is optional to check\n",
    "# if an index file exist, then it will load it\n",
    "# if not, it will rebuild it\n",
    "\n",
    "import os\n",
    "from llama_index import VectorStoreIndex, StorageContext, load_index_from_storage\n",
    "from llama_index import load_index_from_storage\n",
    "\n",
    "if not os.path.exists(\"./sentence_index\"):\n",
    "    sentence_index = VectorStoreIndex.from_documents(\n",
    "        [document], service_context=sentence_context\n",
    "    )\n",
    "\n",
    "    sentence_index.storage_context.persist(persist_dir=\"./sentence_index\")\n",
    "else:\n",
    "    sentence_index = load_index_from_storage(\n",
    "        StorageContext.from_defaults(persist_dir=\"./sentence_index\"),\n",
    "        service_context=sentence_context\n",
    "    )\n",
    "\n",
    "if not os.path.exists(\"./sentence_index_en\"):\n",
    "    sentence_index_en = VectorStoreIndex.from_documents(\n",
    "        [document_en], service_context=sentence_context_en\n",
    "    )\n",
    "\n",
    "    sentence_index_en.storage_context.persist(persist_dir=\"./sentence_index_en\")\n",
    "else:\n",
    "    sentence_index_en = load_index_from_storage(\n",
    "        StorageContext.from_defaults(persist_dir=\"./sentence_index_en\"),\n",
    "        service_context=sentence_context_en\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 创建后处理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 `MetadataReplacementPostProcessor` 类创建了一个后处理器实例，设置了目标元数据键为 `window`。该后处理器的作用是替换目标元数据键的内容。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.indices.postprocessor import MetadataReplacementPostProcessor\n",
    "\n",
    "postproc = MetadataReplacementPostProcessor(\n",
    "    target_metadata_key=\"window\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 `NodeWithScore` 类，将原始节点列表中的每个节点与一个分数关联，形成带分数的节点列表。  \n",
    "使用 `deepcopy` 函数创建了原始节点列表的深度拷贝，以便后续比较。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.schema import NodeWithScore\n",
    "from copy import deepcopy\n",
    "\n",
    "scored_nodes = [NodeWithScore(node=x, score=1.0) for x in nodes]\n",
    "nodes_old = [deepcopy(n) for n in nodes]\n",
    "\n",
    "scored_nodes_en = [NodeWithScore(node=x, score=1.0) for x in nodes_en2]\n",
    "nodes_old_en = [deepcopy(n) for n in nodes_en2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "吧台. \n",
      "bar. \n"
     ]
    }
   ],
   "source": [
    "print(nodes_old[1].text)\n",
    "print(nodes_old_en[1].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用后处理器的 `postprocess_nodes` 方法，替换了带分数的节点列表中目标元数据键的内容。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "replaced_nodes = postproc.postprocess_nodes(scored_nodes)\n",
    "replaced_nodes_en = postproc.postprocess_nodes(scored_nodes_en)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "你好.  吧台.  猫狗.  老鼠\n",
      "hello.  bar.  cat.  dog. \n"
     ]
    }
   ],
   "source": [
    "print(replaced_nodes[1].text)\n",
    "print(replaced_nodes_en[1].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 增设重新排序块"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 `SentenceTransformerRerank` 类创建了一个后处理器实例，设置了参数 `top_n` 为 2，以及使用的模型为 \"BAAI/bge-reranker-base\"。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.indices.postprocessor import SentenceTransformerRerank\n",
    "\n",
    "# BAAI/bge-reranker-base\n",
    "# link: https://huggingface.co/BAAI/bge-reranker-base\n",
    "rerank = SentenceTransformerRerank(\n",
    "    top_n=2, model=\"BAAI/bge-reranker-base\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "创建了一个包含查询文本的 `QueryBundle` 对象，该查询文本为 \"我想要只狗.\"。  \n",
    "创建了一个包含两个带分数的节点的列表，这些节点分别表示包含 \"这是只猫\" 和 \"这是只狗\" 文本的文本节点，分数分别为 0.6 和 0.4。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index import QueryBundle\n",
    "from llama_index.schema import TextNode, NodeWithScore\n",
    "\n",
    "query = QueryBundle(\"我想要只狗.\")\n",
    "\n",
    "scored_nodes = [\n",
    "    NodeWithScore(node=TextNode(text=\"这是只猫\"), score=0.6),\n",
    "    NodeWithScore(node=TextNode(text=\"这是只狗\"), score=0.4),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index import QueryBundle\n",
    "from llama_index.schema import TextNode, NodeWithScore\n",
    "\n",
    "query_en = QueryBundle(\"I want a dog.\")\n",
    "\n",
    "scored_nodes_en = [\n",
    "    NodeWithScore(node=TextNode(text=\"This is a cat\"), score=0.6),\n",
    "    NodeWithScore(node=TextNode(text=\"This is a dog\"), score=0.4),\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 `SentenceTransformerRerank` 类的 `postprocess_nodes` 方法，对带分数的节点列表进行重新排名，考虑到查询文本。重新排名的节点将基于预训练的句子转换模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "reranked_nodes = rerank.postprocess_nodes(\n",
    "    scored_nodes, query_bundle=query\n",
    ")\n",
    "\n",
    "reranked_nodes_en = rerank.postprocess_nodes(\n",
    "    scored_nodes_en, query_bundle=query_en\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "输出了重新排名后的节点列表中的文本和分数。这里展示了句子转换模型对节点重新排名的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('这是只狗', 0.9660425), ('这是只猫', 0.06396222)]\n",
      "[('This is a dog', 0.9182736), ('This is a cat', 0.0014040753)]\n"
     ]
    }
   ],
   "source": [
    "print([(x.text, x.score) for x in reranked_nodes])\n",
    "print([(x.text, x.score) for x in reranked_nodes_en])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 运行索引引擎"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 `as_query_engine` 方法将 `sentence_index` 转换为查询引擎对象 `sentence_window_engine`。  \n",
    "在这里，设置了相似性`（similarity）`的 `top k` 为 6，并传入了 `node_postprocessors` 参数，其中包含了之前创建的 `postproc` 和 `rerank` 后处理器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_window_engine = sentence_index.as_query_engine(\n",
    "    similarity_top_k=6, node_postprocessors=[postproc, rerank]\n",
    ")\n",
    "\n",
    "sentence_window_engine_en = sentence_index_en.as_query_engine(\n",
    "    similarity_top_k=6, node_postprocessors=[postproc, rerank]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用查询引擎的 `query` 方法执行了一个查询，查询的内容是 \"在人工智能领域建功立业的关键是什么?\"。查询引擎将使用之前设置的后处理器进行节点后处理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "window_response = sentence_window_engine.query(\n",
    "    \"在人工智能领域建功立业的关键是什么?\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "window_response_en = sentence_window_engine_en.query(\n",
    "    \"What are the keys to building a career in AI?\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 `LLAMA` 框架提供的 `display_response` 函数展示了查询的响应结果。这通常包括与查询匹配的一组节点，以及它们的文本、分数等信息。  \n",
    "这种方式可以在`Notebook`环境中更好地可视化和理解查询的结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**`Final Response:`** 在人工智能领域建功立业的关键是系统能够正确解释外部数据，从中学习，并利用这些知识通过灵活适应实现特定目标和任务的能力。"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from llama_index.response.notebook_utils import display_response\n",
    "\n",
    "display_response(window_response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**`Final Response:`** Learning foundational technical skills, working on projects, finding a job, and being part of a supportive community are the keys to building a career in AI."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from llama_index.response.notebook_utils import display_response\n",
    "\n",
    "display_response(window_response_en)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二、合并上述操作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`documents`: 要构建索引的文档列表。  \n",
    "`llm`: OpenAI 语言模型实例。  \n",
    "`embed_model`: 嵌入模型的名称或路径。  \n",
    "`sentence_window_size`: 句子窗口的大小。  \n",
    "`save_dir`: 持久化索引的目录。  \n",
    "  \n",
    "创建一个句子窗口的节点解析器（node_parser）。  \n",
    "创建一个包含语言模型和节点解析器等上下文信息的 ServiceContext。  \n",
    "如果指定的目录中不存在索引，则创建一个基于提供的文档的 VectorStoreIndex 并将其持久化到指定目录。  \n",
    "如果目录中已存在索引文件，则从文件中加载索引。  \n",
    "返回构建的句子窗口索引。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from llama_index import ServiceContext, VectorStoreIndex, StorageContext\n",
    "from llama_index.node_parser import SentenceWindowNodeParser\n",
    "from llama_index.indices.postprocessor import MetadataReplacementPostProcessor\n",
    "from llama_index.indices.postprocessor import SentenceTransformerRerank\n",
    "from llama_index import load_index_from_storage\n",
    "\n",
    "\n",
    "def build_sentence_window_index(\n",
    "    documents,\n",
    "    llm,\n",
    "    embed_model=\"local:BAAI/bge-small-zh-v1.5\",\n",
    "    sentence_window_size=3,\n",
    "    save_dir=\"sentence_index\",\n",
    "):\n",
    "    # create the sentence window node parser w/ default settings\n",
    "    node_parser = SentenceWindowNodeParser.from_defaults(\n",
    "        window_size=sentence_window_size,\n",
    "        window_metadata_key=\"window\",\n",
    "        original_text_metadata_key=\"original_text\",\n",
    "    )\n",
    "    sentence_context = ServiceContext.from_defaults(\n",
    "        llm=llm,\n",
    "        embed_model=embed_model,\n",
    "        node_parser=node_parser,\n",
    "    )\n",
    "    if not os.path.exists(save_dir):\n",
    "        sentence_index = VectorStoreIndex.from_documents(\n",
    "            documents, service_context=sentence_context\n",
    "        )\n",
    "        sentence_index.storage_context.persist(persist_dir=save_dir)\n",
    "    else:\n",
    "        sentence_index = load_index_from_storage(\n",
    "            StorageContext.from_defaults(persist_dir=save_dir),\n",
    "            service_context=sentence_context,\n",
    "        )\n",
    "\n",
    "    return sentence_index\n",
    "\n",
    "def build_sentence_window_index_en(\n",
    "    documents_en,\n",
    "    llm,\n",
    "    embed_model=\"local:BAAI/bge-small-en-v1.5\",\n",
    "    sentence_window_size=3,\n",
    "    save_dir=\"sentence_index_en\",\n",
    "):\n",
    "    # create the sentence window node parser w/ default settings\n",
    "    node_parser = SentenceWindowNodeParser.from_defaults(\n",
    "        window_size=sentence_window_size,\n",
    "        window_metadata_key=\"window\",\n",
    "        original_text_metadata_key=\"original_text\",\n",
    "    )\n",
    "    sentence_context = ServiceContext.from_defaults(\n",
    "        llm=llm,\n",
    "        embed_model=embed_model,\n",
    "        node_parser=node_parser,\n",
    "    )\n",
    "    if not os.path.exists(save_dir):\n",
    "        sentence_index_en = VectorStoreIndex.from_documents(\n",
    "            documents_en, service_context=sentence_context\n",
    "        )\n",
    "        sentence_index_en.storage_context.persist(persist_dir=save_dir)\n",
    "    else:\n",
    "        sentence_index_en = load_index_from_storage(\n",
    "            StorageContext.from_defaults(persist_dir=save_dir),\n",
    "            service_context=sentence_context,\n",
    "        )\n",
    "\n",
    "    return sentence_index_en\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`sentence_index`: 已构建的句子窗口索引。  \n",
    "`similarity_top_k`: 相似性查询的 top k。  \n",
    "`rerank_top_n`: 重新排名的 top n。  \n",
    "  \n",
    "定义了两个后处理器：`postproc` 用于替换元数据键，`rerank` 用于使用句子转换模型重新排名节点。  \n",
    "创建一个查询引擎 `sentence_window_engine`，将句子窗口索引转换为查询引擎，并使用定义的后处理器。  \n",
    "返回构建的查询引擎。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_sentence_window_query_engine(\n",
    "    sentence_index, similarity_top_k=6, rerank_top_n=2\n",
    "):\n",
    "    # define postprocessors\n",
    "    postproc = MetadataReplacementPostProcessor(target_metadata_key=\"window\")\n",
    "    rerank = SentenceTransformerRerank(\n",
    "        top_n=rerank_top_n, model=\"BAAI/bge-reranker-base\"\n",
    "    )\n",
    "\n",
    "    sentence_window_engine = sentence_index.as_query_engine(\n",
    "        similarity_top_k=similarity_top_k, node_postprocessors=[postproc, rerank]\n",
    "    )\n",
    "    return sentence_window_engine\n",
    "\n",
    "def get_sentence_window_query_engine_en(\n",
    "    sentence_index_en, similarity_top_k=6, rerank_top_n=2\n",
    "):\n",
    "    # define postprocessors\n",
    "    postproc = MetadataReplacementPostProcessor(target_metadata_key=\"window\")\n",
    "    rerank = SentenceTransformerRerank(\n",
    "        top_n=rerank_top_n, model=\"BAAI/bge-reranker-base\"\n",
    "    )\n",
    "\n",
    "    sentence_window_engine_en = sentence_index_en.as_query_engine(\n",
    "        similarity_top_k=similarity_top_k, node_postprocessors=[postproc, rerank]\n",
    "    )\n",
    "    return sentence_window_engine_en"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "调用之前定义的 `build_sentence_window_index` 函数，传入文档列表、语言模型实例和保存目录，以构建句子窗口索引。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms import OpenAI\n",
    "\n",
    "index = build_sentence_window_index(\n",
    "    [document],\n",
    "    llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n",
    "    save_dir=\"./sentence_index\",\n",
    ")\n",
    "\n",
    "index_en = build_sentence_window_index_en(\n",
    "    [document_en],\n",
    "    llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n",
    "    save_dir=\"./sentence_index_en\",\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "调用之前定义的 `get_sentence_window_query_engine` 函数，传入构建的句子窗口索引和相似性 `top k`，以获取句子窗口的查询引擎。  \n",
    "在这里，`similarity_top_k` 设置为 6。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine = get_sentence_window_query_engine(index, similarity_top_k=6)\n",
    "query_engine_en = get_sentence_window_query_engine(index_en, similarity_top_k=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 三、TruLens评测"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从名为 'generated_questions.text' 的文件中读取生成的问题，将其存储在 `eval_questions` 列表中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_questions = []\n",
    "with open('data/generated_questions.txt', 'r') as file:\n",
    "    for line in file:\n",
    "        # Remove newline character and convert to integer\n",
    "        item = line.strip()\n",
    "        eval_questions.append(item)\n",
    "\n",
    "eval_questions_en = []\n",
    "with open('data/generated_questions_en.txt', 'r') as file:\n",
    "    for line in file:\n",
    "        # Remove newline character and convert to integer\n",
    "        item = line.strip()\n",
    "        eval_questions.append(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义了一个函数 `run_evals`，该函数接受生成的问题列表、`TruLens` 记录器和查询引擎作为参数。对于每个问题，使用 `TruLens` 记录器开始记录，然后使用查询引擎执行查询。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "from trulens_eval import Tru\n",
    "\n",
    "def run_evals(eval_questions, tru_recorder, query_engine):\n",
    "    for question in eval_questions:\n",
    "        with tru_recorder as recording:\n",
    "            response = query_engine.query(question)\n",
    "\n",
    "\n",
    "def run_evals_en(eval_questions_en, tru_recorder, query_engine):\n",
    "    for question in eval_questions_en:\n",
    "        with tru_recorder as recording:\n",
    "            response = query_engine.query(question)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 Tru 类的 `reset_database` 方法重置 `TruLens` 数据库。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🦑 Tru initialized with db url sqlite:///default.sqlite .\n",
      "🛑 Secret keys may be written to the database. See the `database_redact_keys` option of `Tru` to prevent this.\n"
     ]
    }
   ],
   "source": [
    "from utils import get_prebuilt_trulens_recorder\n",
    "\n",
    "from trulens_eval import Tru\n",
    "\n",
    "tru = Tru()\n",
    "\n",
    "tru.reset_database()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 滑窗尺寸设置为1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "调用之前定义的函数 `build_sentence_window_index` 和 `get_sentence_window_query_engine`, 分别构建了句子窗口索引和查询引擎。这里设置了窗口大小为 1，并指定了保存目录为 \"sentence_index_1\"。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_index_1 = build_sentence_window_index(\n",
    "    documents,\n",
    "    llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n",
    "    embed_model=\"local:BAAI/bge-small-zh-v1.5\",  # \"local:BAAI/bge-small-en-v1.5\" for english\n",
    "    sentence_window_size=1,\n",
    "    save_dir=\"sentence_index_1\",\n",
    ")\n",
    "sentence_window_engine_1 = get_sentence_window_query_engine(\n",
    "    sentence_index_1\n",
    ")\n",
    "tru_recorder_1 = get_prebuilt_trulens_recorder(\n",
    "    sentence_window_engine_1,\n",
    "    app_id='sentence window engine 1'\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_index_1_en = build_sentence_window_index_en(\n",
    "    documents_en,\n",
    "    llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n",
    "    embed_model=\"local:BAAI/bge-small-en-v1.5\",  # \"local:BAAI/bge-small-en-v1.5\" for english\n",
    "    sentence_window_size=1,\n",
    "    save_dir=\"sentence_index_1_en\",\n",
    ")\n",
    "sentence_window_engine_1_en = get_sentence_window_query_engine(\n",
    "    sentence_index_1_en\n",
    ")\n",
    "tru_recorder_1_en = get_prebuilt_trulens_recorder(\n",
    "    sentence_window_engine_1_en,\n",
    "    app_id='sentence window engine 1_en'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "调用之前定义的评估函数 `run_evals`，传入生成的问题列表、`TruLens` 记录器 `tru_recorder_1` 和构建的查询引擎 `sentence_window_engine_1`，运行评估任务。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "openai request failed <class 'openai.RateLimitError'>=Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-3.5-turbo in organization org-me3Y2JVoMQFvYW4UUurcFXXM on tokens per min (TPM): Limit 60000, Used 58250, Requested 1999. Please try again in 249ms. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}. Retries remaining=3.\n",
      "openai request failed <class 'openai.RateLimitError'>=Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-3.5-turbo in organization org-me3Y2JVoMQFvYW4UUurcFXXM on tokens per min (TPM): Limit 60000, Used 59832, Requested 502. Please try again in 334ms. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}. Retries remaining=3.\n",
      "openai request failed <class 'openai.RateLimitError'>=Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-3.5-turbo in organization org-me3Y2JVoMQFvYW4UUurcFXXM on tokens per min (TPM): Limit 60000, Used 58502, Requested 1922. Please try again in 424ms. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}. Retries remaining=3.\n",
      "openai request failed <class 'openai.RateLimitError'>=Error code: 429 - {'error': {'message': 'Rate limit reached for gpt-3.5-turbo in organization org-me3Y2JVoMQFvYW4UUurcFXXM on tokens per min (TPM): Limit 60000, Used 59610, Requested 1723. Please try again in 1.333s. Visit https://platform.openai.com/account/rate-limits to learn more.', 'type': 'tokens', 'param': None, 'code': 'rate_limit_exceeded'}}. Retries remaining=3.\n"
     ]
    }
   ],
   "source": [
    "run_evals(eval_questions, tru_recorder_1, sentence_window_engine_1)\n",
    "run_evals(eval_questions_en, tru_recorder_1, sentence_window_engine_1_en)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "tru_recorder_1 = get_prebuilt_trulens_recorder(\n",
    "    sentence_window_engine_1,\n",
    "    app_id='sentence window engine 1'\n",
    ")\n",
    "\n",
    "tru_recorder_1_en = get_prebuilt_trulens_recorder(\n",
    "    sentence_window_engine_1_en,\n",
    "    app_id='sentence window engine 1_en'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "records, feedback = tru.get_records_and_feedback(app_ids=[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>app_id</th>\n",
       "      <th>app_json</th>\n",
       "      <th>type</th>\n",
       "      <th>record_id</th>\n",
       "      <th>input</th>\n",
       "      <th>output</th>\n",
       "      <th>tags</th>\n",
       "      <th>record_json</th>\n",
       "      <th>cost_json</th>\n",
       "      <th>perf_json</th>\n",
       "      <th>ts</th>\n",
       "      <th>Answer Relevance</th>\n",
       "      <th>Context Relevance</th>\n",
       "      <th>Groundedness</th>\n",
       "      <th>Answer Relevance_calls</th>\n",
       "      <th>Context Relevance_calls</th>\n",
       "      <th>Groundedness_calls</th>\n",
       "      <th>latency</th>\n",
       "      <th>total_tokens</th>\n",
       "      <th>total_cost</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>sentence window engine 1</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_87b8d0d554e7d74fa19c16f4692d69cf</td>\n",
       "      <td>\"\\u4eba\\u5de5\\u667a\\u80fd\\u4e2d\\u7684\\u5148\\u9...</td>\n",
       "      <td>\"\\u5148\\u9a8c\\u77e5\\u8bc6\\u5728\\u4eba\\u5de5\\u6...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_87b8d0d554e7d74fa19...</td>\n",
       "      <td>{\"n_requests\": 0, \"n_successful_requests\": 0, ...</td>\n",
       "      <td>{\"start_time\": \"2024-03-12T10:30:49.113462\", \"...</td>\n",
       "      <td>2024-03-12T10:31:02.269094</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.40</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>[{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're...</td>\n",
       "      <td>[{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're...</td>\n",
       "      <td>[{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能  - ...</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>sentence window engine 1</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_b9425d9aa02130eec6c73f7cc6f700f8</td>\n",
       "      <td>\"\\u4eba\\u5de5\\u667a\\u80fd\\u7684\\u81ea\\u6211\\u6...</td>\n",
       "      <td>\"The self-updating and self-improving capabili...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_b9425d9aa02130eec6c...</td>\n",
       "      <td>{\"n_requests\": 0, \"n_successful_requests\": 0, ...</td>\n",
       "      <td>{\"start_time\": \"2024-03-12T10:31:02.914647\", \"...</td>\n",
       "      <td>2024-03-12T10:31:09.293573</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.60</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>[{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人...</td>\n",
       "      <td>[{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人...</td>\n",
       "      <td>[{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能  - ...</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>sentence window engine 1</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_8c266922b3864f14c5aa3fd4fc98923c</td>\n",
       "      <td>\"\\u7ba1\\u7406\\u8005\\u5982\\u4f55\\u7ba1\\u7406AI\\...</td>\n",
       "      <td>\"Management should consider adjusting their wo...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_8c266922b3864f14c5a...</td>\n",
       "      <td>{\"n_requests\": 0, \"n_successful_requests\": 0, ...</td>\n",
       "      <td>{\"start_time\": \"2024-03-12T10:31:09.505494\", \"...</td>\n",
       "      <td>2024-03-12T10:31:12.333074</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.25</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>[{'args': {'prompt': '管理者如何管理AI？', 'response':...</td>\n",
       "      <td>[{'args': {'prompt': '管理者如何管理AI？', 'response':...</td>\n",
       "      <td>[{'args': {'source': '任何的科技都会有瓶颈， 摩尔定律 到⽬前也遇到相...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>sentence window engine 1</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_48957156666710cd55d5059c9ed69b56</td>\n",
       "      <td>\"\\u5f3a\\u4eba\\u5de5\\u667a\\u80fd\\u662f\\u4ec0\\u4...</td>\n",
       "      <td>\"\\u5f3a\\u4eba\\u5de5\\u667a\\u80fd\\u662f\\u4e00\\u7...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_48957156666710cd55d...</td>\n",
       "      <td>{\"n_requests\": 0, \"n_successful_requests\": 0, ...</td>\n",
       "      <td>{\"start_time\": \"2024-03-12T10:31:12.872050\", \"...</td>\n",
       "      <td>2024-03-12T10:31:21.471832</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.80</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>[{'args': {'prompt': '强人工智能是什么？', 'response': ...</td>\n",
       "      <td>[{'args': {'prompt': '强人工智能是什么？', 'response': ...</td>\n",
       "      <td>[{'args': {'source': 'The Behavioral and Brain...</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>sentence window engine 1</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_c3d563072ae3ef443e6e102c53e1677e</td>\n",
       "      <td>\"\\u4eba\\u5de5\\u667a\\u80fd\\u88ab\\u6ee5\\u7528\\u5...</td>\n",
       "      <td>\"The misuse of artificial intelligence can lea...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_c3d563072ae3ef443e6...</td>\n",
       "      <td>{\"n_requests\": 0, \"n_successful_requests\": 0, ...</td>\n",
       "      <td>{\"start_time\": \"2024-03-12T10:31:21.670386\", \"...</td>\n",
       "      <td>2024-03-12T10:31:28.646002</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.50</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>[{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons...</td>\n",
       "      <td>[{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons...</td>\n",
       "      <td>[{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能  - ...</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     app_id  \\\n",
       "0  sentence window engine 1   \n",
       "1  sentence window engine 1   \n",
       "2  sentence window engine 1   \n",
       "3  sentence window engine 1   \n",
       "4  sentence window engine 1   \n",
       "\n",
       "                                            app_json  \\\n",
       "0  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "1  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "2  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "3  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "4  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "\n",
       "                                                type  \\\n",
       "0  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "1  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "2  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "3  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "4  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "\n",
       "                                      record_id  \\\n",
       "0  record_hash_87b8d0d554e7d74fa19c16f4692d69cf   \n",
       "1  record_hash_b9425d9aa02130eec6c73f7cc6f700f8   \n",
       "2  record_hash_8c266922b3864f14c5aa3fd4fc98923c   \n",
       "3  record_hash_48957156666710cd55d5059c9ed69b56   \n",
       "4  record_hash_c3d563072ae3ef443e6e102c53e1677e   \n",
       "\n",
       "                                               input  \\\n",
       "0  \"\\u4eba\\u5de5\\u667a\\u80fd\\u4e2d\\u7684\\u5148\\u9...   \n",
       "1  \"\\u4eba\\u5de5\\u667a\\u80fd\\u7684\\u81ea\\u6211\\u6...   \n",
       "2  \"\\u7ba1\\u7406\\u8005\\u5982\\u4f55\\u7ba1\\u7406AI\\...   \n",
       "3  \"\\u5f3a\\u4eba\\u5de5\\u667a\\u80fd\\u662f\\u4ec0\\u4...   \n",
       "4  \"\\u4eba\\u5de5\\u667a\\u80fd\\u88ab\\u6ee5\\u7528\\u5...   \n",
       "\n",
       "                                              output tags  \\\n",
       "0  \"\\u5148\\u9a8c\\u77e5\\u8bc6\\u5728\\u4eba\\u5de5\\u6...    -   \n",
       "1  \"The self-updating and self-improving capabili...    -   \n",
       "2  \"Management should consider adjusting their wo...    -   \n",
       "3  \"\\u5f3a\\u4eba\\u5de5\\u667a\\u80fd\\u662f\\u4e00\\u7...    -   \n",
       "4  \"The misuse of artificial intelligence can lea...    -   \n",
       "\n",
       "                                         record_json  \\\n",
       "0  {\"record_id\": \"record_hash_87b8d0d554e7d74fa19...   \n",
       "1  {\"record_id\": \"record_hash_b9425d9aa02130eec6c...   \n",
       "2  {\"record_id\": \"record_hash_8c266922b3864f14c5a...   \n",
       "3  {\"record_id\": \"record_hash_48957156666710cd55d...   \n",
       "4  {\"record_id\": \"record_hash_c3d563072ae3ef443e6...   \n",
       "\n",
       "                                           cost_json  \\\n",
       "0  {\"n_requests\": 0, \"n_successful_requests\": 0, ...   \n",
       "1  {\"n_requests\": 0, \"n_successful_requests\": 0, ...   \n",
       "2  {\"n_requests\": 0, \"n_successful_requests\": 0, ...   \n",
       "3  {\"n_requests\": 0, \"n_successful_requests\": 0, ...   \n",
       "4  {\"n_requests\": 0, \"n_successful_requests\": 0, ...   \n",
       "\n",
       "                                           perf_json  \\\n",
       "0  {\"start_time\": \"2024-03-12T10:30:49.113462\", \"...   \n",
       "1  {\"start_time\": \"2024-03-12T10:31:02.914647\", \"...   \n",
       "2  {\"start_time\": \"2024-03-12T10:31:09.505494\", \"...   \n",
       "3  {\"start_time\": \"2024-03-12T10:31:12.872050\", \"...   \n",
       "4  {\"start_time\": \"2024-03-12T10:31:21.670386\", \"...   \n",
       "\n",
       "                           ts  Answer Relevance  Context Relevance  \\\n",
       "0  2024-03-12T10:31:02.269094               0.9               0.40   \n",
       "1  2024-03-12T10:31:09.293573               1.0               0.60   \n",
       "2  2024-03-12T10:31:12.333074               0.8               0.25   \n",
       "3  2024-03-12T10:31:21.471832               0.8               0.80   \n",
       "4  2024-03-12T10:31:28.646002               0.9               0.50   \n",
       "\n",
       "   Groundedness                             Answer Relevance_calls  \\\n",
       "0      1.000000  [{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're...   \n",
       "1      0.800000  [{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人...   \n",
       "2      1.000000  [{'args': {'prompt': '管理者如何管理AI？', 'response':...   \n",
       "3      0.333333  [{'args': {'prompt': '强人工智能是什么？', 'response': ...   \n",
       "4      1.000000  [{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons...   \n",
       "\n",
       "                             Context Relevance_calls  \\\n",
       "0  [{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're...   \n",
       "1  [{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人...   \n",
       "2  [{'args': {'prompt': '管理者如何管理AI？', 'response':...   \n",
       "3  [{'args': {'prompt': '强人工智能是什么？', 'response': ...   \n",
       "4  [{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons...   \n",
       "\n",
       "                                  Groundedness_calls  latency  total_tokens  \\\n",
       "0  [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能  - ...       13             0   \n",
       "1  [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能  - ...        6             0   \n",
       "2  [{'args': {'source': '任何的科技都会有瓶颈， 摩尔定律 到⽬前也遇到相...        2             0   \n",
       "3  [{'args': {'source': 'The Behavioral and Brain...        8             0   \n",
       "4  [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能  - ...        6             0   \n",
       "\n",
       "   total_cost  \n",
       "0         0.0  \n",
       "1         0.0  \n",
       "2         0.0  \n",
       "3         0.0  \n",
       "4         0.0  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "records.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 滑窗尺寸设为3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "调用之前定义的函数，构建了句子窗口索引、查询引擎和 `TruLens` 记录器。这里设置了窗口大小为 3，并指定了保存目录为 \"sentence_index_3\"。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_index_3 = build_sentence_window_index(\n",
    "    documents,\n",
    "    llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n",
    "    embed_model=\"local:BAAI/bge-small-zh-v1.5\",  # \"local:BAAI/bge-small-en-v1.5\" for english\n",
    "    sentence_window_size=3,\n",
    "    save_dir=\"sentence_index_3\",\n",
    ")\n",
    "sentence_window_engine_3 = get_sentence_window_query_engine(\n",
    "    sentence_index_3\n",
    ")\n",
    "\n",
    "tru_recorder_3 = get_prebuilt_trulens_recorder(\n",
    "    sentence_window_engine_3,\n",
    "    app_id='sentence window engine 3'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_index_3_en = build_sentence_window_index_en(\n",
    "    documents_en,\n",
    "    llm=OpenAI(model=\"gpt-3.5-turbo\", temperature=0.1),\n",
    "    embed_model=\"local:BAAI/bge-small-en-v1.5\",  # \"local:BAAI/bge-small-en-v1.5\" for english\n",
    "    sentence_window_size=3,\n",
    "    save_dir=\"sentence_index_3_en\",\n",
    ")\n",
    "sentence_window_engine_3_en = get_sentence_window_query_engine(\n",
    "    sentence_index_3_en\n",
    ")\n",
    "\n",
    "tru_recorder_3_en = get_prebuilt_trulens_recorder(\n",
    "    sentence_window_engine_3_en,\n",
    "    app_id='sentence window engine 3_en'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "调用 `run_evals` 函数，传入生成的问题列表 `eval_questions`、`TruLens` 记录器 `tru_recorder_3` 和构建的查询引擎 `sentence_window_engine_3`，运行评估任务。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_evals(eval_questions, tru_recorder_3, sentence_window_engine_3)\n",
    "run_evals(eval_questions_en, tru_recorder_3_en, sentence_window_engine_3_en)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "records, feedback = tru.get_records_and_feedback(app_ids=[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>app_id</th>\n",
       "      <th>app_json</th>\n",
       "      <th>type</th>\n",
       "      <th>record_id</th>\n",
       "      <th>input</th>\n",
       "      <th>output</th>\n",
       "      <th>tags</th>\n",
       "      <th>record_json</th>\n",
       "      <th>cost_json</th>\n",
       "      <th>perf_json</th>\n",
       "      <th>ts</th>\n",
       "      <th>Answer Relevance</th>\n",
       "      <th>Context Relevance</th>\n",
       "      <th>Groundedness</th>\n",
       "      <th>Answer Relevance_calls</th>\n",
       "      <th>Context Relevance_calls</th>\n",
       "      <th>Groundedness_calls</th>\n",
       "      <th>latency</th>\n",
       "      <th>total_tokens</th>\n",
       "      <th>total_cost</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>sentence window engine 1</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_e9815da1c66c0943f4d155a06f94e9c4</td>\n",
       "      <td>\"\\u4eba\\u5de5\\u667a\\u80fd\\u4e2d\\u7684\\u5148\\u9...</td>\n",
       "      <td>\"\\u5148\\u9a8c\\u77e5\\u8bc6\\u5728\\u4eba\\u5de5\\u6...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_e9815da1c66c0943f4d...</td>\n",
       "      <td>{\"n_requests\": 0, \"n_successful_requests\": 0, ...</td>\n",
       "      <td>{\"start_time\": \"2024-03-10T22:55:51.162029\", \"...</td>\n",
       "      <td>2024-03-10T22:56:03.820730</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're...</td>\n",
       "      <td>[{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're...</td>\n",
       "      <td>[{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能  - ...</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>sentence window engine 1</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_140932cb020d95e0554ecf0489eb42f2</td>\n",
       "      <td>\"\\u4eba\\u5de5\\u667a\\u80fd\\u7684\\u81ea\\u6211\\u6...</td>\n",
       "      <td>\"The self-updating and self-improving capabili...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_140932cb020d95e0554...</td>\n",
       "      <td>{\"n_requests\": 0, \"n_successful_requests\": 0, ...</td>\n",
       "      <td>{\"start_time\": \"2024-03-10T22:56:04.085254\", \"...</td>\n",
       "      <td>2024-03-10T22:56:10.348414</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.8</td>\n",
       "      <td>[{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人...</td>\n",
       "      <td>[{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人...</td>\n",
       "      <td>[{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能  - ...</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>sentence window engine 1</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_5bd9fa7d1b1997c136b9d1d4ce3d2684</td>\n",
       "      <td>\"\\u7ba1\\u7406\\u8005\\u5982\\u4f55\\u7ba1\\u7406AI\\...</td>\n",
       "      <td>\"Management should consider adjusting their wo...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_5bd9fa7d1b1997c136b...</td>\n",
       "      <td>{\"n_requests\": 0, \"n_successful_requests\": 0, ...</td>\n",
       "      <td>{\"start_time\": \"2024-03-10T22:56:10.552787\", \"...</td>\n",
       "      <td>2024-03-10T22:56:13.694339</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.9</td>\n",
       "      <td>[{'args': {'prompt': '管理者如何管理AI？', 'response':...</td>\n",
       "      <td>[{'args': {'prompt': '管理者如何管理AI？', 'response':...</td>\n",
       "      <td>[{'args': {'source': '任何的科技都会有瓶颈， 摩尔定律 到⽬前也遇到相...</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>sentence window engine 1</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_cd9a2aa2bdf60288d1b04cdbeac630f3</td>\n",
       "      <td>\"\\u5f3a\\u4eba\\u5de5\\u667a\\u80fd\\u662f\\u4ec0\\u4...</td>\n",
       "      <td>\"\\u5f3a\\u4eba\\u5de5\\u667a\\u80fd\\u662f\\u4e00\\u7...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_cd9a2aa2bdf60288d1b...</td>\n",
       "      <td>{\"n_requests\": 0, \"n_successful_requests\": 0, ...</td>\n",
       "      <td>{\"start_time\": \"2024-03-10T22:56:13.882046\", \"...</td>\n",
       "      <td>2024-03-10T22:56:21.948222</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.7</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[{'args': {'prompt': '强人工智能是什么？', 'response': ...</td>\n",
       "      <td>[{'args': {'prompt': '强人工智能是什么？', 'response': ...</td>\n",
       "      <td>[{'args': {'source': 'The Behavioral and Brain...</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>sentence window engine 1</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.query_engine....</td>\n",
       "      <td>record_hash_736205619e133e5333d36a19a29293fe</td>\n",
       "      <td>\"\\u4eba\\u5de5\\u667a\\u80fd\\u88ab\\u6ee5\\u7528\\u5...</td>\n",
       "      <td>\"The misuse of artificial intelligence can lea...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_736205619e133e5333d...</td>\n",
       "      <td>{\"n_requests\": 0, \"n_successful_requests\": 0, ...</td>\n",
       "      <td>{\"start_time\": \"2024-03-10T22:56:22.140156\", \"...</td>\n",
       "      <td>2024-03-10T22:56:28.835570</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>[{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons...</td>\n",
       "      <td>[{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons...</td>\n",
       "      <td>[{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能  - ...</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     app_id  \\\n",
       "0  sentence window engine 1   \n",
       "1  sentence window engine 1   \n",
       "2  sentence window engine 1   \n",
       "3  sentence window engine 1   \n",
       "4  sentence window engine 1   \n",
       "\n",
       "                                            app_json  \\\n",
       "0  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "1  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "2  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "3  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "4  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "\n",
       "                                                type  \\\n",
       "0  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "1  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "2  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "3  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "4  RetrieverQueryEngine(llama_index.query_engine....   \n",
       "\n",
       "                                      record_id  \\\n",
       "0  record_hash_e9815da1c66c0943f4d155a06f94e9c4   \n",
       "1  record_hash_140932cb020d95e0554ecf0489eb42f2   \n",
       "2  record_hash_5bd9fa7d1b1997c136b9d1d4ce3d2684   \n",
       "3  record_hash_cd9a2aa2bdf60288d1b04cdbeac630f3   \n",
       "4  record_hash_736205619e133e5333d36a19a29293fe   \n",
       "\n",
       "                                               input  \\\n",
       "0  \"\\u4eba\\u5de5\\u667a\\u80fd\\u4e2d\\u7684\\u5148\\u9...   \n",
       "1  \"\\u4eba\\u5de5\\u667a\\u80fd\\u7684\\u81ea\\u6211\\u6...   \n",
       "2  \"\\u7ba1\\u7406\\u8005\\u5982\\u4f55\\u7ba1\\u7406AI\\...   \n",
       "3  \"\\u5f3a\\u4eba\\u5de5\\u667a\\u80fd\\u662f\\u4ec0\\u4...   \n",
       "4  \"\\u4eba\\u5de5\\u667a\\u80fd\\u88ab\\u6ee5\\u7528\\u5...   \n",
       "\n",
       "                                              output tags  \\\n",
       "0  \"\\u5148\\u9a8c\\u77e5\\u8bc6\\u5728\\u4eba\\u5de5\\u6...    -   \n",
       "1  \"The self-updating and self-improving capabili...    -   \n",
       "2  \"Management should consider adjusting their wo...    -   \n",
       "3  \"\\u5f3a\\u4eba\\u5de5\\u667a\\u80fd\\u662f\\u4e00\\u7...    -   \n",
       "4  \"The misuse of artificial intelligence can lea...    -   \n",
       "\n",
       "                                         record_json  \\\n",
       "0  {\"record_id\": \"record_hash_e9815da1c66c0943f4d...   \n",
       "1  {\"record_id\": \"record_hash_140932cb020d95e0554...   \n",
       "2  {\"record_id\": \"record_hash_5bd9fa7d1b1997c136b...   \n",
       "3  {\"record_id\": \"record_hash_cd9a2aa2bdf60288d1b...   \n",
       "4  {\"record_id\": \"record_hash_736205619e133e5333d...   \n",
       "\n",
       "                                           cost_json  \\\n",
       "0  {\"n_requests\": 0, \"n_successful_requests\": 0, ...   \n",
       "1  {\"n_requests\": 0, \"n_successful_requests\": 0, ...   \n",
       "2  {\"n_requests\": 0, \"n_successful_requests\": 0, ...   \n",
       "3  {\"n_requests\": 0, \"n_successful_requests\": 0, ...   \n",
       "4  {\"n_requests\": 0, \"n_successful_requests\": 0, ...   \n",
       "\n",
       "                                           perf_json  \\\n",
       "0  {\"start_time\": \"2024-03-10T22:55:51.162029\", \"...   \n",
       "1  {\"start_time\": \"2024-03-10T22:56:04.085254\", \"...   \n",
       "2  {\"start_time\": \"2024-03-10T22:56:10.552787\", \"...   \n",
       "3  {\"start_time\": \"2024-03-10T22:56:13.882046\", \"...   \n",
       "4  {\"start_time\": \"2024-03-10T22:56:22.140156\", \"...   \n",
       "\n",
       "                           ts  Answer Relevance  Context Relevance  \\\n",
       "0  2024-03-10T22:56:03.820730               0.9                0.4   \n",
       "1  2024-03-10T22:56:10.348414               1.0                0.5   \n",
       "2  2024-03-10T22:56:13.694339               0.8                0.3   \n",
       "3  2024-03-10T22:56:21.948222               0.9                0.7   \n",
       "4  2024-03-10T22:56:28.835570               0.9                0.3   \n",
       "\n",
       "   Groundedness                             Answer Relevance_calls  \\\n",
       "0           1.0  [{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're...   \n",
       "1           0.8  [{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人...   \n",
       "2           0.9  [{'args': {'prompt': '管理者如何管理AI？', 'response':...   \n",
       "3           1.0  [{'args': {'prompt': '强人工智能是什么？', 'response': ...   \n",
       "4           0.0  [{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons...   \n",
       "\n",
       "                             Context Relevance_calls  \\\n",
       "0  [{'args': {'prompt': '人工智能中的先验知识是如何被存储的？', 're...   \n",
       "1  [{'args': {'prompt': '人工智能的自我更新和自我提升是否可能导致其脱离人...   \n",
       "2  [{'args': {'prompt': '管理者如何管理AI？', 'response':...   \n",
       "3  [{'args': {'prompt': '强人工智能是什么？', 'response': ...   \n",
       "4  [{'args': {'prompt': '人工智能被滥用带来的危害？', 'respons...   \n",
       "\n",
       "                                  Groundedness_calls  latency  total_tokens  \\\n",
       "0  [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能  - ...       12             0   \n",
       "1  [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能  - ...        6             0   \n",
       "2  [{'args': {'source': '任何的科技都会有瓶颈， 摩尔定律 到⽬前也遇到相...        3             0   \n",
       "3  [{'args': {'source': 'The Behavioral and Brain...        8             0   \n",
       "4  [{'args': {'source': '2/2/24, 2:43 PM ⼈⼯智能  - ...        6             0   \n",
       "\n",
       "   total_cost  \n",
       "0         0.0  \n",
       "1         0.0  \n",
       "2         0.0  \n",
       "3         0.0  \n",
       "4         0.0  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "records.head()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "d2l",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
